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ABSTRACT 

Current available robotic systems provide limited support for CAD-based model-driven 
visualization, sensing algorithm development and integration, and automated graphical 
ni^ ing sy f stems - T* 11 * P a P er describes ongoing work which provides the functionality 
necessary to apply advanced robotics to automated manufacturing and assembly 
operations. An interface has been built which incorporates 6-DOF tactile manipulation^ 
displays for three dimensional graphical models, and automated tracking functions which 
I"" automated ma chme vision. A set of tools for single and multiple focal plane 
-!? age pro ? e 1 ssin J’ l and understanding has been demonstrated which utilizes object 
°£ nit i on The resulting tool will enable sensing and planning from 

computationally simple graphical objects. v ° 

A synergistic interplay between human and operator vision is thus created from a 
pro^ammable feedback received from the controller. This approach can be used as the 
sis for implementing enhanced safety in automated robotics manufacturing, assembly 
repair and inspection tasks in both ground and space applications. Thus an interactive 
capability has been developed to match the modeled environment to the real task 
environment for safe and predictable task execution. 

IDENTIFYING MANUFACTURING PROBLEMS 

In many manufacturing facilities, the Ergonomics Coordinator and Engineering Staff 
^aee routme reporting mechanisms for plant production problem reporting and 
OSHA compliance safety reporting. On a monthly basis, issues with product quality and 
mjury incidence are accumulated and reviewed at the plant level (Figure 4 1). These 
problems have been ranked and potential near-term solutions are proposed for safe and 
efficient operator-robot interface. v 

tTSS prodaction and safe }y P lant problems have no immediate solution 

with hard automation or changes in methodology or even workcell redesign These 

applications are candidates for combined operator and robotic solutions For such 
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Figure 1. In-Plant Problem Identification Process 

applications, a standard, rigorous methodology can be followed which consists of the 
following steps 1 : 


( 1 ) 


Identifying ergonomics problem areas which cannot be corrected through 
application of proven technology or job redesign — this is performed throug 
sanctioned plant reporting/ tracking processes outlined in Figure 1. 


(2) Scoping of the problem so estimated resources can be adequately weighed against 
the priority of the resulting solution to plant operation (i.e. preliminary cost benefit 

analysis). 

(3a) Analyzing relevant state of the art - benchmarking the best of currently available 
job analysis/redesign methodology, automated system options, and mixed 
automation and human-in-the-loop methods. 


(3b) Detailing the current job methodology or process to act as a baseline for 
improvement benchmarking 


lThis methodology is derived from the standard Ford Motor Company "Steps to Process Improvement" 
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(4) Envisioning and implementing alternative teleoperated system concepts which 
maximize worker productivity (i.e. operation cycle times), and remain consistent 
with ergonomic principles. 

( 5 ) •^ 1 ° 3 ^ n a bj ) n ' plant testing of most cost effectiv e and superior system concept proposed 

(6) Implementing the benchmarking of the system(s) pilot tested in (5) against prior 

manual practices as captured in step (3b). This benchmarking activity documents 
reduction in worker injury potential, documents any improvements in worker 
satisfaction and productivity, and documents any improvements in product quality 
if applicable. n J 

(7) Continuous improvement of commercialization of successful human-in-the-loop 
technology applications. Because (5) generates an unequivocal business case, 
commercialization can proceed expeditiously. 

Through the above process applications have been identified that can benefit from 
telerobotic technology. These applications are focused on making the workplace more 
safe for the factory worker, and at the same time improve efficiency by enhanced human- 
robot interaction for task generation and environment calibration. This reduction of injury 
aspect of this form of automation is advantageous to both worker and management The 
project development process used (Figure 2) has been effective in getting customers 
acceptance and support for this technology. 



Figure 2. Technology Development Process 
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Telerobotics can remove a machine operator from hazardous environments by making it 
possible to understand, and tune operator's actions through the control interfaces to 
procude a specific effect at the remote machine. This added awamess is enhanced by 
combining computer generated visual or graphical cues with computer generated 
tactile/force feedback cues. Towards that goal an interactive computer vision-robot 
handcontroller for safe automated flexible manufacturing has been developed based on 
Cybernet 6-DOF force reflection handcontroller (see Figure 3), and advanced machine 
vision processing. 


6 DOF HANDCONTROLLER 

The PER-Force handcontroller manipulates robots or objects by "feel." Simulating a 
"sense of touch" by "force-reflection" with a wide motion range greatly enhances the 
efficiency of operations which require manipulation and dynamic control of objects m 
multidimensional spaces. The PER-Force handcontroller is a small backdrivable robot 
which moves in 6 degrees of freedom, 3 linear positions (x-, y-, z-) and 3 attitudes (roll, 
pitch, yaw) [1]. 



Figure 3. The Cybernet PER-Force 6 DOF Force Reflection Handcontroller 
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An operator can use this motorized handle to precisely position other robots or 
graphically displayed objects to a given location (x-, y-, z-> and tool angle (roll, pitch, 
yaw). This is done by a host computer or a robot control system that reads the 
handcontroller joint or transformed position, velocity, or force. 

"Force-feedback" can be generated on each axis by the handcontroller through 6 small, 
brushless, DC servo motors. The six axis force-reflection output and six axis orientation 
and position control makes the manipulation and "feeling" of multidimensional objects or 
datasets extremely easy. The kinematic arrangement of the PER-Force stick is design for 
maximum simplicity and performance for both the electronic digital servo process and 
mechanical gravity compensation (Figure 4). The first two stages are a simple X-Y table 
(driven by a rack and pinion, and held in place by two parallel rails per stage). By 
convention X is side to side and Y is back and forth. Because these axes work parallel to 
gravity, no compensation is required. 




Figure 4. Handcontroller Kinematic Arrangement 


The next stage is the Z axis, which is translated up and down. This axis levitates the yaw, 
pitch, and roll mechanisms, and the structures to which they attach. However, the Z 
motor and gear train themselves do not levitate (thus saving additional weight). The Z 
stage is gravity compensated by two constant force springs which are matched to the 
upper stage weight. The first revolute stage is yaw, which operates parallel to the base 
and therefore needs no gravity compensation. The next axis is pitch. The last axis is roll. 
All six axes of motion intersect at a point through the middle of the handle. We have 
found this to be the most comfortable pivot point for teleoperation. 

The PER-Force Handcontroller is completely programmable including the time interval 
associated with the servo loop. Within the device, a servo shell process begins by 
initializing the interrupt loop data structures and linkages. After this is completed, the 
interrupt process runs every clock interval to provide the servo routines with a 
deterministic time base. In the PC-AT version, this clock interrupt is a re-programmed 
use of the system clock/timer. 
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The user initializes the desired time interval for the servo loop (shorter for crisper 
response -- longer to accommodate longer interrupt processing loops). The timer 
interrupt is skipped until enough ticks have been seen to represent this programmed 
interval. Then the interrupt level servo logic is executed. When the servo loop begins to 
run, it first collects each encoder value, computes estimates for velocity and acceleration, 
and then computes and option set of translations, scales, and rotations on the XYZ data 
and the pitch, roll, yaw data. This global transform allows the entire servo function space 
to be rotated, shifted, or scaled to create different force field "views". 2 

For a typical mas ter-slave protocol, the input consists of slave positions or forces which 
are transformed from native slave coordinates to scaled master Cartesian coordinates (and 
then uses them to update gains, center locations, or forces in one or more interrupt level 
servo functions to create a force "feel"). Because the user actually installs pointers to 
their own control and/or command code, complete flexibility is available from the PER- 
Force servo structure (or course many useful controls are already included in the 
libraries). This flexibility enables the development of advanced user interfaces which use 
force feedback to implement new forms of machine-operator cooperative problem 
solving. 

Towards integrating the 6-DOF handcontroller into the SGI environment, a stream 
module was written to generate handcontroller motion and button events for the SGI. The 
stram module converts a device specific data stram into an independent representation 
which the server interprets. 


THE OPERATOR CONTROL STATION 

The approach for the operator control station (OCS) is to develop a hybrid man/machine 
system, based on the competitive advantage of both the human and the computer, which 
will allow supervised control of a remote telerobot from an OCS which communicates 
with the telerobot over a communications channel that has a latency of several seconds 
and a thruput limited to several megabits per second. The operator control station [2] 
represents the local site of a local-remote architecture telerobotic system for remote 
operations. The designed architecture supports multiple local-site operator control 
stations with a common remote site task execution system [31 as shown in Figure 5. The 
operator interface of the local operator control station has two primary parts: perception 
and manipulation. Perception provides an interactive means for modeling the remote site 
scene. Manipulation provides interactive task description, simulation, editing, and 
execution. Central to the operator interface is the knowledge base which holds 
information on the state of the local and remote site systems and manipulation and 
perception data. The methodology of the local-remote system is to build and simulate 
manipulation and sensing commands on the local site, using a model of the robot and its 
environment stored in the knowledge base, which has been updated and validated with 
feedback sensory data. 


2 This is analogous to changing the view port to a 3D model in model 3D graphics 
engines. 
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Figure 5. Local Remote Telerobot Control Architecture 


The control station is hosted on a Silicon Graphics IRIS 310 VGX Power Series 
workstation, equipped with a 6-axis "handcontroller" input device, and with LCD 
shuttered glasses for 3-D stereo viewing. The local site software is written in C, utilizing 
X Windows, Motif, the IRIS Inventor graphics product, and a small library of X resource 
manager extensions called the Widget Creation Library, developed at JPL. 

The interactive perception module is designed to use a combination of operator input and 
machine vision to refine and calibrate the model of the task environment that resides in 
the knowledge base. Interactive perception utilizes computer power for precision 
measurement, and human perception for recognition, scene segmentation, and rough 
location designation, where reliable and efficient computer algorithms are unavailable. To 
aid human perception, the system provides views from multiple video cameras, including 
a stereo view for depth perception. 3-D graphics is overlayed onto both the stereo video 
views and the monocular video views in either wireframe, transparent, or solid. The three 
primary functions of perception are object localization, object model editing, and camera 
localization. In object localization, the operator translates and operates the graphics- 
overlay until reasonable registration has been achieved with video images of the object 
from multiple viewpoints. In camera localization, the operator uses the handcontroller to 
adjust the graphics overlay on a video image to best register the overlay against some 
visible objects whose position is accurately calibrated with respect to each other. For 
object model editing, the operator uses the handcontroller to move a 3-D cursor in order 
to designate the 3-D positions of vertices, and connect them graphically with edges.. 

The interactive task description capability is to make task description verification, and 
execution as simple as possible to the operator. This is achieved by providing the operator 
with a library of skills which the remote manipulators can perform. Skills are generic 
motion types, e.g., guarded-motion, move- to- touch, hinge, slide, screw, insert etc. When 
parameterized, a skill becomes a command which can be sent to the remote site for 
execution. 
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Figure 6. OCS Software Block Diagram 


As shown in Figure 6, the software that implements the operator control station may be 
conceptualized in three levels. Operator interface modules directly interacting with the 
operator. The graphical user interface (GUI) allows the operator to command desired 
activities and provides high-level sequencing of subtasks, calling perception and 
manipulation modules as needed. 

The current state of implementation of the local control station has enabled evaluation of 
the system's performance on several different tasks. It has been found to be an effective, 
convenient interface for controlling a telerobot in the presence of time delay in a semi- 
structured environment. 


INTEGRATING THE TWO ENVIRONMENTS 

The focus of our work has been directed at integrating the Per-Force into the OCS to 
enable rapid implementation of teleoperation environment within the manufacturing 
environment. Towards this end, we are developing an environment which features a 6 
DOF force reflection handcontroller, contemporary CAD and graphics enviroimients, 
image processing, and standardized robotic platform interfaces, to produce a 6 DOF robot 
controller (see Figure 7). 
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Figure 7. Combined Architecture 

The development of such an environment is important because it facilitates more 
dynamic utilization of robots in manufacturing applications It provides Aeon«S»^Sh 
access lofc geometry and physical properties of the iTtainSjjS 
thegap between simulation and teleoperation. In doing so it provides an interesting 
vehicle for providing the human operator with supervision skill? by combining human 

SifSfS"* V1S1 ° n ca Pf^ llties - . Within system, object feature descriptions ntS 
w?/ thC g j°T etnc , a descriptions of parts (which link the feature locations to obiect 
ocations and the object locations to gripping points) to enable machine vkinn 
registration of graphical part descriptions with physical sensed real-world information 

ShSh^ 15 reglstratl ° n has been made, the operator can manipulate parts in a virtualreahtv 
which causes part placement within the real world. F eauty 

APPLICATION TO MANUFACTURING 

The process flow described above for identifying manufacturing applications has been 

l m r an H UfaCtUri ? g a PPhcations of die system und^r devdopment *££ 
process has identified several applications for this technology. One such application 
involves loading transmission cases within the Ford Motor Company Cureend v 

S Z S Z S * n CaSCS areclell '' ered ^ the plant in large bins (nominally 4'xTx 7'rnTz Y 
(Figure 8) in an unordered state and must be loaded onto kitting fixtures in a standard 
orientation. These cases weigh over 50 lbs and am now loaded 

repem.ve motion injuries Standard manual lift assist devices T in'convTnfen “and 
cumbersome enough that those performing this task have not adopted their consistent use. 

characteristic of one type of previously identified operation in which 
supervised teleoperation technology may be beneficial. That is it is representative nf 
tasks which require lifting heavier than safe loads. These m’sks arfstifl 2 i J 
performed because part of the operation requires flexibility (i.e. rapid accommodation to 
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different transmission case types) and precision placement (placement in a standard 
orientation onto a standard kitting fixture for insertion onto the transfer line). This seim- 
“S bin picking operation has been studied for full automation, usually by 
computer vision-guided robotics, for many years. This approach has resisted solution 
because of the complexity of part motion and computer vision program changeover for 
Sdh suc£s“ve new part (even relatively small part design changes dictate new computer 
vision recognition and part gripping strategies). 


CASING DIMENSIONS 
(IN IN.) 


BIN DIMENSIONS 
<IN FT.) 



< 2 ? 


CASINO WEIGHT = 40 LBS. (APPROX.) 

Figure 8. Transmission Cases and Their Transport Bins 

Our approach combines an automated pick and place cycle robotics system with specific 
steps where an operator takes teleoperated control of the system. It includes an industrial 
robotic arm with an end effector capable of gripping and manipulating transmission 
cases Control and manipulation is provided by a telerobotic force reflecting 
handcontroller electronically interfaced to the robot arm. Both teleoperated and 
automatic motions are supporting in the system to achieve both the flexibility of 
teleoperation coupled with the rapid cycle speed possible through automation The 
alignment of transmission cases is enhanced through operator views provided by two 
Cameras (and a graphics enhanced video viewing system). Transmission allotments are 
provided through a conveyor system. All moving parts/robotic elements are surrounded 
by safety fences. Figure 9 illustrates such a system. 


Figure 10 shows the basic architecture of the design. The physical concept is shown in 
Figure 11a (top) and lib (side). The unit consists of a conveyor subsystem for 
conveying work bins to the unloading site, a robot arm which is controlled to pick-up the 
transmission cases and place them on the assembly line conveyor system, and an operator 
control station from which operations are directed. The operator control station 
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Figure 9. Conceptual Layout of the Transmission Bin-Picking Cell 

(Figure 9) consists of a telerobotic handcontroller for operator control of the robot 
system and a video viewing system, which allows transmission cases to be aligned for 
appropriate placement on the assembly transfer line. 



Figure 10. System architecture 
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As currently envisioned, a heavy duty conveyer system will be installed to convey the 
work bins into the work envelope. The conveyor must be designed to allow three full 
bins to be brought into the work envelope before the empty bins need to be removed. 
This system is gravity fed with safety brakes controlling the flow of the bins. The initial 
dimensions are estimated at 40 feet by 6.5 feet. Each bin is estimated to weigh 2000 lbs. 



The robotic arm which actually replaces manual lifting needs an envelope range of 
nominally eleven feet. A system in this size class has payload capacity of approximately 
220 lbs. The arm is controlled by a combination of automated controls and a manually 
operated force reflecting telerobotic handcontroller. An appropriate transmission case- 
gripping end-effector is part of the robot system. The robot system is caged for operator 
safety. 

The basic control interface to the operator is a telerobotic handcontroller with active force 
feedback. This allows the operator to have complete real-time control of the robot system 
position, orientation, and end-effector state (open/closed) and can allow the operator to 
feel robot-casing collisions and contacts. This handcontroller technology base also 
includes the technology of the robot system controller which acts as the cell control 
system, which drives conveyor, robot, and handcontroller actuators simultaneously. 

A video viewing system is used to provide the operator with the visual cues necessary to 
properly align the transmission casing for the final placement. As currently envisioned, 
the operator cues the system to bring the case into a pre-specified location, within view of 
the video cameras, and performs manual alignment to a graphic overlaid on top of the 
casing video display. The cameras provide two orthogonal views which enhance the 
operator's ability to orient the casing prior to an automated step which places the case 
onto its suspension pins (on the carriers which transport the case to down stream 
assembly operations using the transmission conveyor system). 
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Figure 1 lb. Teleoperated Transmission Case Bin Picking Cell (Side) 

Table 1 show a typical supervisor machine cycle for the conceptual teleoperated 
transmission bin-picking cell. The operator controls all actions of the robot arm through 
the use of a telerobotic handcontroller and control buttons. The operator is located 
slightly above the bin to provide direct visual contact for case grasping/gripping 
operations. Automation moves the robot to a location above the expected location of the 
next case within the bin (if this "guess" is incorrect the operator can control the robot to 
the correct location in the next step manually). Control is then passed to the operator 
(through the 6 axis handcontroller). The operator moves the robot arm to a casing using 
teleoperation (The operator controls the robot's end-effector by moving the 
handcontroller stick handle in the direction/orientation needed to mate the end-effector 
with the case; Tactile "feel" is provided by the stick to the operator’s hand and visual 
cues are provide by direct viewing of the end-effector and the workpiece; End-effector 
grip is closed/opened through a trigger). Once the case is seized a control button is 
pressed which initiates a pre-programmed motion to move the transmission case to the 
alignment station (within view of alignment cameras). At the alignment station the 
operator visually aligns the case, using two cameras and a monitor (alignment is 
physically controlled by moving the force reflecting handcontroller to properly line up 
the video image of the unit with a graphics overlay target). After alignment has been 
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Table 1. Basic Machine Cycle: Diagram of Operations 
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Manual 
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Button Press 
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zone to the operator (control is exercised 
through the handcontroller) 


Manual 


Operator manipulates the handcontroller to 
place robot into gripping position around a 
case. 


Hand 

Controller 

Operation 


Manual 


Operator cues end-effector gripping 
operation 


Trigger Press 


Automated 


End-effector closes 


Manual 
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viewing station (if grip is successful) 
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Button Press 
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fixturing and places the case on the line 

Automated Robot moves back to a pick-up zone in the 
bin - operation cycles 
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achieved, another switch depression initiates an automated action which places the casing 
on the transmission holder of the assembly line transfer conveyor. 

Automated robotic pick-and-place operations can be much faster that the equivalent 
manual tasks, especially if the object to be manipulated is heavy for the operator. Thus, 
the cycle outlined decreases the loading operation time and at the same time retains 
manual flexibility to adapt to new transmission designs (assuming the hand-offs between 
man and machine are properly defined and tested). 

Our system has the potential for implementing such solutions, and for quantifying capital 
cost and payback over an extended factory operational lifetime. The development of such 
applications and the verification of this payback is important to future human-in-the-loop 
manufacturing robotics technology. 
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